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SUMMARY 

The  importance  of  uninhabited  vehicles  in  military  environments  has  been  recognised  for  many  years. 
Much  progress  has  been  made  in  fields  such  as  robotics  and  data  communications,  but  not  so  much  on  the 
human  factors  issues.  This  is  partly  due  to  the  trend  to  strive  for  ever  higher  levels  of  system  autonomy, 
leaving  the  operator  the  task  of  supervising  the  system.  However,  by  increasing  system  automation,  the 
cognitive  system  of  operators  may  become  a  bottleneck.  A  recent  concept  (which  we  call  tele-presence)  to 
tackle  this  problem  is  to  design  man-machine  interfaces  that  allow  operators  to  optimally  use  their 
perceptual  motor  system  in  order  to  relieve  the  cognitive  system.  Such  an  interface  would  enable  sheer 
effortless  looking  and  moving  around  in  the  remote  environment  by  bringing  the  operator  back  in  the  front 
of  the  loop,  resulting  in  a  redistribution  of  task  demands  from  the  cognitive  level  to  the  perceptual  level.  A 
successful  implementation  would  result  in  increased  situational  awareness  and  reduced  cognitive  load. 
This  paper  starts  with  describing  the  theoretical  background  behind  tele-presence  concept  among  others 
based  on  Endsley  ’s  Situational  Awareness  model.  In  the  second  part,  we  describe  the  design  of  a  tele¬ 
presence  interface  for  controlling  an  unmanned  ground  vehicle  and  an  initial,  exploratory  study.  The 
findings  of  this  experiment  show  that  we  were  not  able  to  elicit  a  robust  tele-presence  effect  yet.  We 
discuss  the  results  in  relation  to  the  present  state  of  technology,  interface  characteristics  such  as  delay 
between  input  and  feedback,  behaviour  and  motion  sickness,  and  make  recommendations  on  future 
research  directions. 


1  INTRODUCTION 

Since  the  introduction  of  remotely  controlled  vehicles,  an  important  trend  has  been  to  apply  as  much 
automation  as  allowed  by  the  state  of  the  art  in  the  relevant  technologies.  An  important  reason  for  this  is 
the  assumption  that  a  human  in  the  loop  is  a  weak  link  in  the  system.  However,  even  nowadays,  the  human 
operator  is  still  part  of  the  system,  partly  to  be  responsible  for  the  actions  of  the  system,  but  also  because 
state  of  the  art  technology  has  not  advanced  enough  to  implement  intelligence  and  full  autonomy.  Because 
of  this  trend,  the  task  of  the  human  operator  has  changed  dramatically  over  the  years  from  an  operator  in 
the  loop  who  has  full  manual  control  over  the  remote  vehicle  to  a  supervisory  controller  who  monitors  the 
status  of  the  vehicle  and  is  allowed  not  more  than  to  provide  instructions  at  a  high  level.  As  Van  Erp  [22] 
argues,  automating  tasks  that  can  be  automated  on  account  of  the  technology  does  not  necessarily  result  in 
better  system  performance.  By  leaving  the  tasks  that  can  not  be  automated  to  the  operator,  he  or  she  ends 
up  with  the  tasks  at  which  humans  may  also  not  be  very  good,  or  a  task  set  that  is  not  well  balanced. 
Furthermore,  by  turning  the  operator  into  a  supervisory  controller,  the  system  may  not  make  optimal  use 
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of  the  human  intelligence,  knowledge,  and  skills.  For  example,  reconnaissance  experts  are  able  to  tell  a 
complete  story  based  on  certain  cues  (like  the  presence  of  a  specific  type  of  vehicle  at  a  specific  location), 
while  an  automated  system  needs  to  search  a  large  area  to  get  a  complete  picture. 

The  advantages  of  remote  sensing  and  acting  are  indisputable,  despite  the  negative  effects  of  taking  the 
operator  out  of  the  loop.  However,  by  bringing  the  human  operator  back  in  the  front  of  the  loop,  we  expect 
that  system  performance  on  certain  tasks  can  be  improved  compared  to  a  (semi-)  autonomous  system  and 
a  supervisory  controller. 

1.1  Theoretical  basis  for  tele-Presence 

Two  theoretical  models  served  as  a  basis  for  the  tele-presence  concept.  The  first  is  related  to  situational 
awareness  (SA),  in  particular  the  model  of  Endsley  [5]  with  three  levels  of  SA,  the  second  is  related  to 
navigation,  in  particular  the  prenav  model  by  Van  Erp  [23]. 


1.1.1  Tele-presence  and  SA 

Although  there  are  numerous  definitions  of  Situation  Awareness  (SA;  see  Van  Erp  [21]  for  an  overview), 
we  will  use  Endsley’ s  [6,  7]:  “Situation  awareness  is  the  perception  of  the  elements  in  the  environment 
within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status 
in  the  near  future”.  Important  themes  of  this  definition  (called  levels  by  Endsley)  are  printed  in  Italic.  It  is 
important  to  notice  that  it  is  only  useful  to  see  SA  in  relation  to  a  specific  situation  -a  set  of  environmental 
conditions  and  system  states  with  which  the  participant  is  interacting-  that  can  be  characterised  uniquely 
by  its  priority  goals  and  response  options.  SA  becomes  meaningless  if  the  operator  needs  to  know 
everything  about  everything.  This  means  that  SA  requirements  will  change  if  the  situation  changes.  For 
remote  control  situations,  SA  is  a  key  element.  However,  acquiring  and  maintaining  SA  becomes 
increasingly  difficult  as  the  complexity  and  dynamics  of  the  environment  increase.  Under  some 
circumstances,  many  decisions  are  required  within  a  fairly  narrow  time  span,  and  task  performance 
requires  an  up-to-date  analysis  of  the  environment.  Because  the  state  of  the  environment  is  constantly 
changing  (often  in  complex  ways)  a  major  portion  of  the  operator's  job  becomes  that  of  obtaining  and 
maintaining  good  SA. 

Level  1  in  Endsley’s  model  (see  Figure  1)  refers  to  the  perception  of  the  elements  in  the  environment  and 
their  relationship  to  other  points  of  reference  (i.e.  internal  model).  At  this  level,  relevant  characteristics 
(colour,  size,  speed  and  location)  and  the  dynamics  of  the  objects  in  the  environment  are  represented.  This 
aspect  is  similar  to  what  Barfield  et  al.  [2]  termed  spatial  awareness.  Level  2  of  SA  goes  beyond  simply 
being  aware  of  the  elements  that  are  present,  and  includes  an  understanding  of  the  significance  of  the 
elements.  Based  on  level  1  knowledge,  the  operator  forms  a  holistic  picture  of  the  environment, 
comprehending  the  significance  of  objects  and  events.  Thus,  the  integration  of  various  level  1  data 
elements  at  level  2  of  SA  is  crucial  for  the  comprehension  of  the  situation.  Finally,  the  ability  to  project 
the  future  status  of  the  elements  in  the  environment  forms  the  third  and  highest  level  of  SA. 

The  critical  level  with  respect  to  the  current  research  is  level  1.  As  Endsley  states,  perception,  including 
pre-attentive  processing  and  short-term  sensory  memory,  is  the  core  construct  of  level  1 .  Furthermore, 
attention,  which  affects  information  sampling  (level  1),  and  can  be  guided  by  goals  and  objectives  (box 
A).  The  fact  is  that  in  many  remote  control  interfaces,  SA  is  not  based  on  perception  and  pre-attentive 
processing  because  the  interface  does  not  support  the  use  of  the  perceptual  motor  system,  but  instead 
requires  cognitive  effort  already  at  level  1 .  By  bringing  the  operator  back  in  the  loop  and  enabling  the  use 
of  the  perceptual  motor  system  to  acquire  and  maintain  adequate  level  1  SA,  cognitive  resources  become 
available  to  enhance  SA  at  higher  levels. 
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Figure  1:  Endsley’s  Model  of  Situational  Awareness  (adapted  from  [20]). 
See  text  for  explanation. 


1.1.2  Tele-presence  and  navigation 

Besides  building  SA,  navigation  (including  vehicle  control)  is  a  relevant  task  in  remote  control.  Prenav 
(see  Figure  2)  is  based  on  an  integration  of  models  for  navigation  and  workload:  Sheridan’s  model  for 
supervisory  (vehicle)  control  [19],  Wickens’  information  processing  model  [30,  31],  Veltman  and  Jansen’s 
workload  framework  [29],  and  Rasmussen’s  framework  of  skill-based,  rule-based  and  knowledge-based 
behaviour  [16,  17].  An  important  loop  in  prenav  is  the  information-processing  loop  (indicated  by  thick 
arrows):  sensation=>perception=>decision=>action,  and  back  via  environment  or  a  display.  The 
perception  and  decision  steps  are  called  the  cognitive  ladder  in  prenav.  The  five  parallel  arrows  as  input  to 
sensation  denote  that  different  modalities  (e.g.,  touch,  vision,  and  audition)  can  be  involved  and  that  the 
processing  in  these  modalities  is  parallel  at  least  up  to  the  sensation  level  (Multiple  Resource  Theory  [30, 
31,  32]).  Contrary  to  many  other  models,  the  information-processing  loop  in  prenav  is  not  a  serial  process 
in  which  all  steps  need  to  be  completed.  Specific  for  prenav  and  important  for  the  tele-presence  concept  is 
the  existence  of  two  shortcuts,  indicated  with  thin  arrows  in  Figure  2.  The  first  is  the  sensation=>action 
shortcut.  When  a  sensation  directly  evokes  an  action,  it  bypasses  the  cognitive  ladder  completely. 
Examples  include  reflexive  or  highly  trained  tasks  such  as  maintaining  our  balance  and  braking  when  a 
child  suddenly  crosses  the  road.  The  second  shortcut  is  the  perception=>action  shortcut.  A  percept  may 
also  directly  result  in  an  action  thus  bypassing  the  decision  process.  This  is  the  case  for  automated 
“if... then”  rules.  For  example  in  driving:  when  you  see  a  stop  sign,  you  decelerate.  This  process  does  not 
involve  a  conscious  decision,  but  does  require  the  interpretation  of  the  visual  information  as  a  stop  sign. 

The  second  loop  in  the  prenav  model  is  based  on  the  workload  framework  of  Veltman  and  Jansen  [29]  that 
stresses  the  role  of  the  state  of  the  operator  on  the  information  processing  loop.  In  the  prenav  model,  the 
operator  state  specifically  affects  the  cognitive  ladder,  but  not  the  sensation=>action  shortcut.  External 
stressors  such  as  sleep  deprivation,  G  load,  vibration,  and  wearing  night  vision  goggles  may  affect  the 
state  of  the  operator. 
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Figure  2:  Prenav,  an  Integrated  Model  for  Navigation  by  Van  Erp.  See  text  for  explanation. 


Both  Endsley’s  SA  model  and  the  prenav  model  have  a  level  at  which  tasks  can  be  performed 
automatically  without  involving  cognition.  This  level  is  based  on  very  powerful  perceptual  motor 
mechanisms  that  have  evolved  over  thousands  of  years  and  are  trained  everyday  when  we  walk  and  look 
around,  or  when  we  drive  a  car  or  bicycle.  To  employ  these  powerful  mechanisms  in  building  SA  and 
navigating,  the  operator  has  to  be  in  the  front  of  the  loop.  As  we  concluded  before,  the  automation  trend  in 
remote  control  sends  the  operator  backwards  in  the  loop  or  even  out  of  the  loop.  The  consequences  are  that 
the  operator  has  to  use  his  cognitive  system  to  perform  tasks  that  in  real  life  would  have  been 
accomplished  at  the  perceptual  motor  level.  The  key  to  this  lies  in  the  design  of  the  man-machine 
interface. 


hypothesised 
positive  effects 

presence  • — ►  better  SA  and  navigation 

?  t  negative  effects 

implemented  by  ;  i  through  operator  state 

natural  moving. _ ^motion 

and  viewing  sickness 

Figure  3:  The  Tele-Presence  Concept  Expects  Positive  Effects  on  SA  and  Navigation 
Performance.  However,  tele-presence  is  implemented  by  an  interface  that  allows 
natural  looking  and  moving  about  which  may  facilitate  motion  sickness. 
Motion  sickness  can  act  as  an  external  stressor  and  affect  the  operator 
state  which  may  negatively  affect  SA  and  navigation  performance. 


1.2  Presence  and  motion  sickness 

There  may  be  one  important  drawback  of  the  tele-presence  concept.  Since  the  operator  is  controlling  the 
motion  of  her  or  himself,  or  that  of  a  remote  ‘sensor  unit’,  there  is  a  possibility  that  motion  sickness  or 
simulator  sickness,  respectively,  will  arise.  Simulator  sickness,  or  motion  sickness  in  general,  arises  when 
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the  (afferent)  information  on  self-motion  as  sensed  by  the  eyes,  the  vestibular  system,  and  the 
somatosensory  system,  differs  from  the  motion  that  is  anticipated  by  the  brain.  Motion  sickness  may 
negatively  affect  the  operator  state  and  as  a  consequence  the  building  of  SA  and  navigation  performance. 
Simulator  sickness  is  a  familiar  problem  to  operators  of  flight-  or  ground-vehicle  simulators.  It  also  occurs 
in  virtual  environments  and  PC  games  such  as  first  person  shooters,  and  is  then  designated 
‘cybersickness’.  Virtual  Environment  (VE)  systems  are  notorious  for  cybersickness  [8,  9,  10].  These 
systems  often  employ  a  visual  display  (e.g.  HMD)  that  is  controlled  by  head  or  body  movements  of  the 
user.  The  VE  user  can  be  considered  a  very  experienced  ‘pilot’  with  a  very  accurate  internal  model  of  his 
own  body  movements,  so  that  differences  in  visual  feedback,  e.g.  a  time  delay,  have  large  consequences 
for  the  perceived  motion,  or,  motion  sickness.  In  other  words,  the  more  closely  a  motion  interface  mimics 
the  own  bodily  movements,  the  more  critical  the  motion  feedback  becomes,  and  the  higher  the  risk  of 
motion  sickness.  In  the  prenav  model,  a  motion  sickness  evoking  situation  is  an  external  stressor  that 
affects  the  operator  state  and  thus  task  performance  (see  Figure  3). 


2  DESIGN  OF  A  REMOTELY  OPERATED  GROUND  VEHICLE 
INTERFACE  THAT  FACILITATES  TELE-PRESENCE 

2.1  Presence  and  man-in-the-loop  interface  design 

The  point  of  departure  is  that  the  interface  should  optimally  comply  with  the  perceptual  motor  system  of 
the  operator  requiring  a  minimum  of  cognitive  effort  to  navigate  and  build  level  1  SA. 


2.1.1  Tele-presence  viewing 

When  looking  around  in  the  real  world  we  have  a  large  field  of  view  at  our  disposal  including  stereoscopic 
and  other  cues,  and  adjust  our  viewing  direction  by  body,  head,  and  eye  movements.  The  quality  of  current 
state  of  the  art  remote  or  indirect  viewing  systems  is  still  far  from  that  of  direct  view  with  respect  to  for 
example  resolution,  field  of  view,  colour  depiction  and  contrast  range  [25].  In  a  remote  control  situation,  a 
further  degradation  may  be  caused  by  bandwidth  limitations.  Even  if  we  had  a  sensor  system  with  the 
specifications  of  the  human  eye,  it  is  not  very  likely  that  the  images  could  actually  be  transported  from  the 
remote  location  to  the  operator. 

Based  on  a  literature  review  [1,  3,  4,  12,  13,  14,  18,  21,  24,  25,  26,  28]  we  identified  the  critical  image 
parameters  for  driving.  These  are:  field  size,  image  minification,  the  number  of  pixels  per  degree  of  field 
of  view,  availability  of  vehicle  reference  points,  image  quality,  and  stereovision  (the  latter  for  off-the-road 
driving  only).  Using  current  CCD  technology  as  point  of  departure,  we  came  to  the  following  system 
design.  First,  a  large  field  of  regard  is  preferred  with  respect  to  both  vehicle  control  and  situation 
awareness.  This  cannot  be  realised  by  using  a  single  camera  with  a  fixed  viewing  direction,  but  only  by  a 
moving  camera  or  by  using  multiple  cameras.  A  moving  camera  has  two  important  advantages.  First,  there 
are  no  artificial  effects  caused  by  the  switching  between  cameras  and  second,  a  moving  camera  eases  the 
introduction  of  stereoscopic  viewing.  To  enable  the  use  of  proprioceptive  cues  on  viewing  direction,  the 
remote  camera  should  preferably  be  head-slaved  and  the  images  should  preferably  be  presented  in  the 
viewing  direction  of  the  operator,  either  by  a  wide  screen  projection  or  by  a  head  mounted  display. 
Because  an  HMD  makes  it  easier  to  implement  stereoscopic  images,  it  may  be  the  preferred  option. 
Although  stereoscopic  images  may  not  be  required  with  respect  to  vehicle  control  on  flat  terrain  they  may 
be  beneficial  in  off  the  road  situations  and  for  manipulation  tasks.  Also,  image  minification  must  be 
avoided,  that  is,  the  field  size  of  the  sensor  should  match  that  of  the  display.  Adding  it  all  together,  a 
system  that  consists  of  a  head-slaved  remote  camera  and  a  head-mounted  display  seems  to  have  the  most 
potential  to  facilitate  vehicle  control,  natural  viewing  and  tele-presence. 
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2.1.2  Tele-presence  locomotion 

Although  the  operator  is  controlling  a  platform  and  not  a  remote  human  being,  walking  may  be  the  most 
natural  way  of  controlling  the  platform’s  speed  and  direction.  However,  this  would  also  restrict  the 
characteristics  of  the  platform  to  the  limits  of  walking  or  running.  This  restriction  will  be  most  outspoken 
in  the  moving  speed  of  the  platform  (i.e.,  in  translations)  and  not  so  much  in  its  direction  (i.e.,  rotations). 
Furthermore,  operators  may  experience  difficulties  in  distinguishing  vehicle  and  camera  rotations. 
Therefore,  the  potential  advantage  of  a  natural  locomotion  may  be  most  pronounced  in  platform  rotations 
and  be  of  less  relevance  to  vehicle  translation  (i.e.  forward  speed).  Maintaining  the  coupling  between  the 
direction  and  rotations  of  the  body  and  those  of  the  platform  may  be  beneficial  for  situation  awareness  and 
also  be  a  more  natural  control  of  the  locomotion  and  thus  be  beneficial  to  the  operator’s  presence.  This 
coupling  can  be  implemented  either  by  measuring  the  body  direction  of  a  standing  operator  that  turns 
around  his  body  axis  or  of  seated  operator  that  rotates  his  chair  like  you  rotate  an  office  chair  around  its 
axis.  It  should  be  noted  that  the  effects  of  whole-body  interaction  techniques  on  way  finding  performance 
in  a  virtual  environment  compared  to  joystick  control  (e.g.  [15])  are  not  very  outspoken  and  probably 
dependent  on  the  amount  of  required  manoeuvring. 

2.2  Research  questions 

In  the  second  part  of  this  paper,  we  describe  an  exploratory  experiment,  designed  to  gain  insight  into  the 
effect  of  tele-presence  viewing  and  tele-presence  locomotion.  Based  on  the  design  issues  above,  we 
developed  three  remote  control  interfaces  for  a  wheeled  vehicle  with  an  on-board  moving  camera  and 
compared  performance  and  behaviour  with  that  of  two  control  conditions  in  which  the  operator  was  on¬ 
board  the  vehicle  (i.e.,  physically  present  in  the  ‘remote’  environment).  The  three  remote  control 
interfaces  were  (with  ROV  referring  to  remotely  operated  vehicle):  [a]  ROY  joystick:  an  interface  with  a 
fixed  monitor  and  two  joysticks:  one  to  control  the  platform  and  one  to  control  the  camera,  this  is  a 
baseline  ROV  interface;  [b]  ROV  HMD:  an  interface  that  enables  tele-presence  viewing,  consisting  of 
head-slaved  camera  control  and  a  head  mounted  display,  while  the  platform  is  controlled  with  a  joystick; 
[c]  ROV  stepping  around:  an  interface  that  enables  tele-presence  viewing  as  in  ROV  HMD  and  tele¬ 
presence  locomotion.  In  this  interface,  heading  changes  (i.e.  rotation  about  the  vertical  axis)  of  the  vehicle 
are  slaved  to  the  heading  changes  of  the  chair  on  which  the  operator  is  seated.  The  operator  can  thus 
change  the  heading  of  the  vehicle  by  stepping  around.  Forward  motion  of  the  platform  is  still  controlled  by 
a  joystick.  The  two  control  conditions  were:  [d]  wheelchair:  with  the  participant  seated  on  the  ROV  with 
unrestricted  viewing;  and  [e]  wheelchair  goggles:  with  the  participant  on  the  ROV  but  wearing  field  size 
restricting  goggles  that  had  a  fieldsize  equivalent  to  that  of  the  remote  viewing  system. 

We  had  the  following  hypotheses:  1)  the  respective  scores  of  remote  control  interface  a,  b,  and  c  will 
increase  in  that  order  on  performance,  presence  and  SA;  2)  interface  c  will  result  in  different  vehicle 
control  behaviour  than  interfaces  b  and  c;  3)  compared  to  the  wheelchair  conditions,  all  three  ROV 
interfaces  may  result  in  occurrences  of  motion  sickness;  4)  the  degree  of  motion  sickness  will  resemble  the 
presence  score  of  the  three  remote  control  interfaces;  5)  SA  scores  will  be  higher  for  the  interfaces  that 
allow  natural  viewing,  i.e.,  b  and  c. 


3  METHODS 

3.1  Participants 

Fourteen  university  students  participated  in  the  experiment,  seven  males  and  seven  females.  The 
participants  were  on  average  22  years  old,  ranging  from  1 8  to  29  years  old.  All  participants  were  in  good 
physical  health.  The  participants  were  right  handed  and  had  good  eyesight,  possibly  corrected  by  contact 
lenses  (participants  wearing  glasses  were  not  selected,  because  of  the  interference  with  the  apparatus 
used). 
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3.2  Apparatus 

The  control  conditions  were  ran  with  the  wheelchair  depicted  in  Figure  4.  In  the  wheelchair  goggles 
condition,  the  participants  wore  the  field  restricting  goggles  depicted  in  Figure  4.  In  the  wheelchair 
condition,  the  participant  had  unrestricted  viewing. 

The  remotely  operated  vehicle  (ROV)  system  consisted  of  two  parts:  the  control  station  and  the  vehicle. 
The  control  station  was  located  in  a  room  adjacent  to  the  driving  course.  The  vehicle  was  based  on  an 
electronic  wheelchair  equivalent  to  the  one  used  in  the  control  conditions.  Two  small  cameras  were 
mounted  on  a  moving  base  with  three  degrees  of  freedom  (pan,  tilt  and  roll)  on  the  front  of  the  vehicle. 
See  Figure  5. 


Figure  4:  Left:  the  Electronic  Wheelchair  Used  in  the  Control  Conditions.  The  joystick  is 
mounted  at  the  end  of  the  right  armrest.  Right:  the  tunnel  vision  goggles  with  the  same 
rectangular  fieldsize  as  the  camera  system  in  the  remote  control  situation. 


Figure  5:  Left:  the  Remotely  Operated  Vehicle  (ROV).  Right:  a  close-up  image  of  the  camera 
system  consisting  of  two  CCD  cameras  and  a  pan-tilt-roll  platform. 
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From  the  control  station,  the  operator  controlled  the  camera  system  and  the  vehicle  itself.  This  was  done 
with  the  following  three  interface  variants.  In  the  ROY  joystick  variant  the  camera  platform’s  heading  and 
pitch  were  controlled  by  a  table  mounted  joystick  with  the  left  hand  with  the  images  of  the  right  camera 
displayed  on  a  table  mounted  CRT  in  front  of  the  operator.  The  direction  and  forward  speed  of  the  vehicle 
were  controlled  with  a  table  mounted  joystick  with  the  right  hand.  In  the  ROY  HMD  variant,  the  cameras 
were  coupled  to  the  head  motions  of  the  operator  (which  were  tracked  with  a  mechanical  head  tracker  in 
three  directions:  pan,  tilt  and  roll)  with  the  images  presented  on  an  HMD.  Vehicle  control  was  as  in  the 
ROV  joystick  variant.  In  the  ROV  stepping  around  variant,  viewing  was  as  in  the  ROV  HMD  variant,  but 
controlling  the  rotations  of  the  vehicle  was  different.  In  this  mode,  the  vehicle  rotations  were  coupled  to 
the  rotations  of  the  chair  of  the  control  station  while  forward  and  backward  speed  was  controlled  with  a 
small  joystick  attached  to  the  chair  on  the  control  station  (see  Figure  6). 


Figure  6:  In  the  ROV  Stepping  around  Control  Condition,  the  Heading  of  the 
Remotely  Operated  Vehicle  is  Coupled  to  the  Heading  of  the  Operator's 
Chair.  Stepping  around  thus  controls  vehicle  heading. 


3.3  Tasks  and  measures 

The  participants  had  to  drive  through  a  course  marked  by  poles  that  allowed  the  testing  of  different  aspects 
of  vehicle  control,  including  lateral,  longitudinal  and  speed  control,  distance  estimation,  positioning  and 
manoeuvring  in  confound  spaces.  Task  performance  was  measured  with  time  to  completion,  number  of 
poles  hit  on  left  and  right  side  and  distance  estimation  errors.  To  measure  SA,  we  used  a  target 
identification  task.  Three  different  A4-sized  target  pictures  were  placed  at  random  locations  in  the  course. 
After  completing  the  course,  participants  had  to  draw  the  memorised  location  and  identity  of  the  targets 
they  had  seen  on  a  floor  plan  of  the  course.  After  the  participants  completed  the  course  and  the  target 
identification,  they  filled  out  a  questionnaire.  The  first  part  of  the  questionnaire  discussed  issues  regarding 
the  presence  of  the  operator,  i.e.  to  what  extent  did  the  operator  feel  present  at  the  location  of  the  vehicle 
(in  the  course).  The  second  part  dealt  with  motion  sickness  issues  (how  did  the  participants  feel  after 
completing  the  course).  The  Misery  Scale  (MISC)  was  used:  an  11 -point  scale  on  which  the  participant 
could  indicate  how  nauseated  they  felt  after  completing  a  condition. 

3.4  Procedures  and  training 

When  the  participants  arrived,  they  received  an  extensive  instruction  about  the  tasks  and  the  experiment. 
The  five  conditions,  the  course,  the  SA  task,  and  the  questionnaires  were  all  explained.  The  participants 
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signed  an  informed  consent.  After  this,  the  experiment  started.  First,  the  participant  was  given  time  to 
practise  driving.  Especially  in  the  remote  conditions,  participants  needed  to  learn  how  to  handle  the 
vehicle  and  how  to  manage  the  HMD.  Furthermore  they  had  to  learn  speed  and  distances.  When  they  were 
able  to  handle  the  vehicle  they  practised  driving  the  course  to  learn  the  sequence  of  the  tasks  in  the  course 
and  practise  them.  Each  participant  completed  all  five  interface  conditions  in  a  within  subjects  design.  The 
order  of  the  conditions  was  balanced  over  the  participants  to  the  extent  possible.  Before  a  participant 
started  with  a  condition  he  or  she  drove  the  course  at  least  one  time  for  practise.  When  they  could  drive  the 
course  faultless,  the  measurements  started.  During  the  training  they  received  feedback  on  performance 
from  the  experimenter.  After  they  completed  the  training,  the  participants  had  a  short  break.  During  that 
break,  the  experimenter  placed  the  three  targets  in  the  course.  Directly  after  finishing  the  course,  the 
participants  completed  the  SA  task  and  filled  out  the  presence  and  MISC  questionnaires.  When  all  the 
forms  were  filled  out,  the  participant  could  begin  with  the  practise  for  the  next  condition  (after  a  break,  if 
necessary).  The  experiment  lasted  a  full  morning  or  afternoon  per  participant. 


4  RESULTS 

4.1  Platform  control 

The  data  on  course  completion  performance  generally  showed  that  performance  in  both  control  conditions 
was  better  than  in  the  three  ROV  conditions  (statistically  significant).  None  of  the  platform  control 
measures  showed  a  difference  between  the  three  ROV  control  interfaces. 

Besides  task  performance  we  were  interested  in  the  control  input  given  in  the  three  ROV  conditions  (i.e. 
vehicle  control  behaviour).  The  raw  results  are  depicted  in  Figure  7.  Inspection  of  the  control  strategy 
showed  that  the  ROV  stepping  around  mode  differed  on  two  important  aspects  form  the  two  joystick 
controlled  modes.  First,  the  maximum  output  (±  5  in  this  plot)  is  not  reached  as  often  as  in  the  other 
conditions.  Second,  there  are  more  points  on  the  x-axis.  This  indicates  that  there  is  often  a  rotation  input 
without  forward  speed. 


rotation  input 


rotation  input 


Figure  7:  The  Rotation  Input  (Horizontal  Axis)  and  Translation  Input  (Vertical  Axis)  in  the  Three 
Remote  Control  Conditions:  left:  ROV  stepping  around,  middle:  ROV  HMD,  right:  ROV  joystick. 


4.2  Situational  Awareness 

We  analysed  the  score  (object  and  location  correct)  and  found  a  main  effect  of  condition:  F(4,  40)  =  3.80, 
p  <  .05.  The  means  are  shown  in  Figure  8.  The  post  hoc  only  revealed  a  significant  difference  between  the 
ROV  stepping  around  mode  and  both  wheelchair  conditions. 
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ORGANIZATION 


condition 


Figure  8:  Scores  (Object  and  Location  Correct)  in  the 
Situation  Awareness  Task  as  Function  of  Condition. 


4.3  Presence  questionnaire 

The  presence  questionnaire  consisted  of  nine  questions,  each  with  an  answer  scale  from  1  to  5.  After 
appropriate  reversing  of  some  of  the  scales  we  performed  an  ANOVA  on  the  mean  scores.  This  revealed  a 
significant  effect  of  condition:  F(4,  40)  =  13.40,  p  <  .01.  The  means  over  all  nine  questions  is  given  in 
Figure  9.  A  higher  score  here  indicates  a  higher  presence.  The  post  hoc  analysis  showed  that  both 
wheelchair  conditions  differed  significantly  from  the  three  ROY  conditions. 


wheelchair  ROV  stepping  around  ROV  joystick 

wheelchair  goggles  ROV  HMD 


condition 


Figure  9:  Mean  Presence  Score  on  a  Scale  from  1  to  5  for  the  Five  Conditions. 
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4.4  Motion  sickness  questionnaire 

In  general,  the  motion  sickness  incidence  was  50%.  Seven  out  of  14  subjects  rated  minimally  a  MISC 
score  of  5  (“nauseated”)  in  one  or  more  conditions,  22  of  all  67  MISC  ratings  (3  scores  were  missing) 
were  higher  than  3  (“stomach  awareness”).  Three  participants  had  to  end  a  condition  prematurely  because 
of  nausea.  Two  of  them  were  able  to  recover  sufficiently  to  continue  with  the  other  conditions.  However, 
one  of  them  suffered  from  severe  nausea  (9)  in  her  fourth  condition,  and  stopped  the  experiment. 


wheelchair  ROV  stepping  around  ROV  joystick 


wheelchair  goggles  ROV  HMD 

condition 


Figure  10:  Change  in  the  MISC  (Misery  Score)  as  Function 
of  Condition.  The  MISC  has  an  1 1  -point  scale. 


Since  motion  sickness  is  a  cumulative  phenomenon,  and  all  conditions  were  performed  within  four  hours, 
simply  looking  at  the  average  MISC  ratings  may  overestimate  the  problems  in  certain  conditions. 
Participants  who  experienced  the  most  provocative  condition  at  the  beginning,  may  have  transferred  their 
discomfort  to  the  next  conditions.  Although  participants  were  given  an  opportunity  to  recover  from 
nausea,  it  is  likely  that  they  became  more  sensitive  in  the  subsequent  conditions.  To  correct  for  this  order 
effect,  we  analysed  the  change  in  MISC  score  for  each  condition.  In  cases  where  the  rating  was  lower  than 
the  rating  in  a  previous  condition,  indicating  that  the  subject  had  partly  recovered,  the  negative  change  was 
set  to  zero.  The  results  of  this  analysis  are  shown  in  Figure  10.  There  was  a  main  effect  for  Condition:  F(4, 
52)  =  6.15,  p  <  .01).  A  Tukey  post-hoc  analysis  indicated  that  the  mean  rating  in  the  ROV  HMD  differed 
from  both  wheelchair  conditions. 

4.5  Remote  control  mode  preference 

Finally,  we  asked  seven  of  the  participants  to  rate  the  ROV  modes  (without  taking  the  wheelchair 
conditions  into  account):  a.  which  mode  did  you  find  best?  b.  Which  one  worst?  And  c.  which  one 
provided  the  best  feeling  of  presence?  This  forced  choice  paradigm  was  thought  to  be  more  sensitive  to 
small  differences  than  the  presence  questionnaire.  The  raw  results  are  depicted  in  Figure  11.  The  graph 
shows  that  the  ROV  stepping  around  and  the  ROV  joystick  mode  were  perceived  as  both  the  best  mode, 
but  also  as  the  worst  mode.  The  ROV  HMD  mode  was  never  perceived  as  the  worst  mode.  The  ROV 
stepping  around  mode  was  for  four  of  the  participants  the  mode  with  the  most  presence. 
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Figure  11:  Preference  Scores  for  the  Three  ROV  Conditions. 


5  DISCUSSION  AND  CONCLUSION 

5.1  Discussion  on  the  present  results 

In  general  the  data  show  that  the  control  conditions  are  better  than  the  ROV  conditions  and  that  there  is  no 
significant  effect  of  ROV  interface  variant.  Although  the  three  ROV  variants  were  designed  to  mediate 
different  levels  of  presence,  these  effects  are  not  apparent  in  the  data.  In  the  Introduction  we  stated  five 
hypotheses  with  respect  to  the  differences  between  the  five  interface  designs  on  presence,  SA  and 
performance. 

The  first  hypothesis  concerned  the  scores  of  the  ROV  conditions  on  performance,  presence  and  SA.  We 
expected  that  the  scores  would  be  lowest  with  the  ROV  joystick  interface,  intermediate  with  the  ROV 
HMD,  and  highest  with  the  ROV  stepping  around  interface.  This  hypothesis  is  not  supported  by  the 
present  data.  Actually,  there  are  no  significant  differences  at  all  between  the  three  ROV  conditions. 

The  second  hypothesis  was  that  the  ROV  stepping  around  mode  would  result  in  different  vehicle  control 
behaviour.  We  did  find  differences  between  the  three  ROV  modes,  indicating  that  the  ROV  stepping 
around  mode  differed  from  the  two  other  ROV  modes.  The  differences  seem  to  indicate  either  more 
cautious  behaviour  or  less  bang-bang  control  in  the  ROV  stepping  around  interface:  the  maximum  output 
is  reached  less  often  and  rotation  is  often  used  without  forward  speed.  Interestingly,  there  is  no 
disadvantage  of  the  ROV  stepping  around  on  task  performance,  including  the  time  to  complete  the  course. 
This  indicates  that  the  observed  control  behaviour  is  probably  caused  by  more  cautious  behaviour.  The 
behaviour  seems  to  reflect  a.  the  fact  that  in  the  ROV  stepping  around,  the  participants  moved  the  vehicle 
more  smoothly  than  in  the  jerky  bang-bang  control  like  behaviour  with  the  other  two  ROV  interfaces,  and 
b.  the  strategy  to  rotate  the  vehicle  to  look  around  in  stead  off  moving  the  cameras.  The  change  in  sensor 
images  of  the  environment  is  equivalent  whether  the  vehicle  and  sensor  platform  rotate  together  with 
respect  to  the  ground  or  whether  the  sensor  platform  rotates  with  respect  to  a  ground- fixed  vehicle.  The 
preference  for  rotating  the  whole  vehicle  over  rotating  the  cameras  only  seems  to  indicate  that  participants 
were  not  very  comfortable  with  using  the  head-slaved  viewing  direction.  The  advantage  of  rotating  the 
vehicle  might  be  the  fact  that  the  sensor  maintains  a  fixed  viewing  direction  with  respect  to  the  vehicle’s 
driving  direction.  A  common  problem  for  operators  is  the  loss  of  camera  direction  with  respect  to  the 
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driving  direction.  The  option  of  a  vehicle  that  rotates  around  its  yaw  axis  provides  the  option  for  operators 
to  look  around  by  rotating  the  vehicle  including  the  sensors  without  loosing  awareness  of  sensor  and  /  or 
vehicle  orientation. 

The  third  and  fourth  hypotheses  concerned  motion  sickness.  We  expected  that  the  three  ROV  modes 
would  result  in  higher  occurrences  of  motion  sickness  than  the  wheelchair  conditions  and  that  the  degree 
of  motion  sickness  would  correlate  with  the  presence  scores  of  the  three  ROV  interfaces.  There  was  a  high 
(50%)  incidence  of  motion  sickness.  This  is  probably  due  to  delays  in  both  the  platform  and  camera 
control  loop  and  the  delay  in  the  image  system  due  to  several  conversion  between  the  cameras  and  the 
HMD.  The  ROV  HMD  and  ROV  stepping  around  modes  showed  the  highest  increase  in  MISC  rating. 
This  confirms  the  provocative  nature  of  the  use  of  an  HMD  as  the  primary  medium  for  orientation  in  a 
virtual  or  remote  world.  After  correcting  for  possible  order  effects,  it  seemed  that  the  ROV  HMD  was  the 
most  provocative.  We  should  note,  however,  that  the  most  susceptible  participant,  who  reached  a  9 
(“retching”)  in  this  condition,  missed  the  ROV  stepping  around  condition.  It  is  likely  that  her  missing  data 
would  have  raised  the  average  rating  in  this  condition  as  well.  On  the  other  hand,  the  participants  probably 
(mis-)  used  the  stepping  around  functionality  to  look  around  therewith  reducing  the  amount  of  head 
rotations.  Based  on  these  data  it  is  difficult  to  decide  whether  active  rotation  about  the  yaw-axis  affected 
the  degree  of  discomfort.  It  should  also  be  noted  that  in  the  stepping  around  mode  the  gain  of  the  yaw- 
motion  of  the  remote  unit  did  not  exactly  equal  1 ,  so  that  its  yaw-rotation  did  not  exactly  match  that  of  the 
operator.  Certainly,  this  may  have  introduced  a  confounding  in  judgements  of  motion  perception  and 
discomfort.  The  judgements  of  presence  and,  more  in  particular,  motion  perception  (data  not  presented  in 
this  paper),  did  not  show  significant  differences  between  the  ROV  conditions.  Therefore  we  cannot 
conclude  as  to  whether  a  higher  sense  of  presence  (we  had  expected  that  this  would  be  the  case  in  the 
ROV  stepping  around  condition)  results  in  higher  sickness  ratings. 

The  fifth  hypothesis  stated  that  the  SA  scores  would  be  higher  for  the  interfaces  that  allow  natural 
viewing,  i.e.  the  ROV  HMD  and  ROV  stepping  around  modes.  This  hypothesis  could  also  not  be 
substantiated  by  the  data.  One  out  of  the  two  conditions  scored  lower  than  the  wheelchair  conditions, 
while  the  other  conditions  didn’t  significantly  differ  from  each  other.  There  may  be  several  causes, 
amongst  others  the  fact  that  the  mechanical  headtracker  imposed  restrictions  on  the  field  of  regard  in  the 
natural  viewing  conditions  as  compared  to  the  ROV  joystick  and  the  wheelchair  modes.  Also,  motion 
sickness  effects  may  have  restricted  the  head  motions  of  the  operators  to  those  necessary  to  build  a  good 
sense  of  SA. 

The  problem  of  the  current  hypotheses  lies  in  the  fact  that  they  are  difficult  to  reject  because  non-effects 
can  be  blamed  on  the  state  of  the  technology  that  is  not  sufficient  to  elicit  tele-presence.  However,  we  can 
conclude  that  the  present  exploratory  experiment  does  not  provide  indications  that  support  the  hypotheses. 
The  disadvantage  of  an  interface  that  allows  natural  viewing  and  motion  control  (i.e.,  the  ROV  stepping 
around  interface),  namely  the  risk  of  motion  sickness,  does  not  outweigh  the  potential  advantages. 
Because  these  potential  advantages  are  very  much  dependent  on  the  state  of  the  technology  it  is  also 
difficult  to  generalise  the  findings.  The  interface  design  was  based  on  a  reasonable  starting  point  with 
respect  to  sensor,  vehicle,  and  bandwidth  technology,  optimised  on  the  basis  of  available  human  factors 
data  on  vehicle  and  camera  control.  The  resulting  interface  was  not  powerful  enough  to  result  in  a  high 
degree  of  presence,  at  least  not  higher  than  in  the  ROV  joystick  mode. 

The  most  important  conclusion  is  therefore  that  we  can  neither  accept  nor  reject  the  proposition  that  a 
man-in-the-loop  interface  results  in  higher  presence,  and  that  higher  presence  results  in  better  performance 
and  SA,  simply  because  we  failed  to  complete  the  first  part:  test  an  interface  design  that  results  in  higher 
presence. 
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Figure  12:  A  Handheld  Telepresence  Interface  that  Allows  the  User  to  Steer 
the  Remote  Camera  by  a  Device  that  Resembles  a  Pair  of  Binoculars. 


5.2  Follow-on  studies 

After  the  initial  study  described  in  this  paper,  we  decided  that  the  system  required  an  upgrade  to  be  able  to 
test  the  tele-presence  concept.  Encouraged  by  the  anecdotal  evidence  we  gathered  in  informal  evaluations 
of  the  set-up  by  military  personal  and  experts  in  tele-operated  devices,  we  implemented  several  upgrades 
to  improve  the  coupling  of  body  and  head  motions  to  vehicle  and  camera  platform  motions,  respectively. 
One  such  improvement  is  to  change  the  analogue  camera  system  that  required  several  conversions 
between  camera  and  HMD  presentation  with  a  digital  system  reducing  the  time  lag  in  the  image  system. 
The  second  upgrade  was  to  replace  the  analog  link  for  vehicle  control  signals  with  a  digital  one  that  also 
allows  to  better  fine  tune  the  relation  between  operator  input  and  vehicle  and  camera  platform  motions 
(e.g.  in  off  set  and  gain).  Furthermore,  we  introduced  artificial  vehicle  reference  points  indicating  the 
heading  direction  of  the  platform  (both  auditory  and  visual  cues).  Finally,  we  replaced  the  HMD  and 
mechanical  head  tacking  system  by  the  hand-held  display  system  depicted  in  Figure  12.  This  system 
senses  the  heading,  pitch  and  roll  of  the  tele-presence  binoculars  and  feeds  them  into  the  camera  system. 
Apart  from  being  portable,  the  system  allows  behaviour  that  resembles  the  use  of  a  pair  of  binoculars  most 
military  personnel  is  accustomed  to.  In  follow-on  experimentation  with  a  ground  control  station  with  the 
updated  data  links  and  image  system  [11],  advantages  of  a  tele-presence  interface  over  tele-operated  and 
autonomous  modes  were  found,  confirming  the  potential  of  the  tele-presence  concept. 

5.3  Conclusions  and  recommendations 

The  tele-presence  concept  claims  that  employing  the  human  perceptual  and  psychomotor  system  for 
navigation  and  to  build  Situational  Awareness  at  the  lowest  level  frees  cognitive  resources  that  are  needed 
to  build  SA  at  higher  levels.  In  a  remote  control  situation,  the  possibility  to  employ  the  perceptual  motor 
system  is  mediated  by  tele-presence  and  the  remote  control  interface:  the  experience  of  being  present  at 
the  remote  location  and  the  possibility  of  using  natural  viewing  and  locomotion.  The  drawback  of  this 
philosophy  is  that  tele-presence  might  also  mediate  motion  sickness:  higher  levels  of  tele-presence  or 
being  immersed  in  the  remote  location  might  make  the  user  more  susceptible  to  a  mismatch  between  the 
actual  sensory  input  and  what  the  brain  expects.  In  the  present  experiment  we  were  not  able  to  elicit  a 
robust  tele-presence  effect. 
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Future  research  directions  include  the  following.  The  human  perception  system  is  not  restricted  to  vision,  a 
logical  expansion  is  to  include  other  senses.  Three-dimensional  audio  and  the  sense  of  touch  seem  to  be 
the  trivial  choices.  In  a  real-life  situation  a  soldier  would  not  want  to  operate  without  auditory  signals, 
while  vibrations  and  forces  are  important  to  provide  feedback  on  for  example  vehicle  behaviour  and 
condition.  Future  systems  may  also  implement  super-human  sensory  systems  such  as  night  vision  and 
NBC  detectors.  Also,  next  to  expanding  the  sensory  side,  acting  in  the  remote  environment  must  go 
beyond  moving  and  looking  around.  A  relevant  example  is  defusing  an  explosive.  Again,  an  interface  that 
allows  natural  behaviour  might  be  able  to  minimise  the  required  cognitive  capacity  of  the  operator  to 
control  tools  and  eventually  result  in  better  performance.  A  third  important  issue  is  the  preferred  level  of 
automation  and  the  tasks  that  are  better  suited  for  high  levels  of  automation  (such  as  transit  form  A  to  B) 
and  tasks  where  tele-presence  has  advantages  (such  as  gathering  intelligence).  Because  a  remotely 
operated  platform  might  not  be  operating  alone  in  the  theatre,  collaboration  between  multiple  uninhabited 
platforms  and  between  uninhabited  platforms  and  soldiers  is  an  important  research  issue,  including  for 
example  collaborative  haptics  and  shared  SA. 
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Introduction 


Automation:  operator  in  the  loop  =>  supervisory  control 
Automation  -t  better  system  performance 

•  Two  parts: 

•  part  1 :  theoretical  background 

•  part  2:  exploratory  study 

Two  models  from  the  Human  Factors  engineering  domain 

•  Situational  awareness:  Endsley 
Steering  &  control:  prenav 
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Endsley’s  SA  model 
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•  abilities 

•  experience 

•  training 


The  core  constructs  of  level  1  SA  are: 

•  perception, 

•  short-term  sensory  memory, 

•  pre-attentive  processing. 
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control 


Summary  part  1 


Both  Endsley's  SA  model  and  the  prenav  model  have  a  level  at 
which  tasks  can  be  performed  automatically  without  involving 
cognition. 

To  employ  these  powerful  mechanisms  in  building  SA  and  in 
vehicle  control,  the  operator  has  to  be  in  the  front  of  the  loop. 

The  key  to  this  lies  in  the  design  of  the  man-machine  interface. 
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Risk 
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positive  effects 

Tele-presence  • — ►better  SA  and  vehicle  control 
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Experiment 


5  interface  variants:  three  remote  control  and  two  present 
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Remote  control 

camera 

vehicle 

control 

control 

joystick 

joystick 

head-slaved 

joystick 

head-slaved 

body-slaved 

Present 

viewing 

vehicle 

full 

joystick 

restricted 

joystick 

Experiment:  task  and  measures 


Presence  (rating  scale) 
SA  (memorising  objects) 
Motion  sickness  (MISC) 
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Conditions 


Wheelchair,  unrestricted  FOV 


Wheelchair,  but  with  fieldsize  restricting  goggles 


Remote;  head-slaved  HMD  and  stepping  around  vehicle  control 


Remote;  head-slaved  HMD  and  joystick  controlled  vehicle 


Remote;  monitor  and  joystick  controlled  camera  and  vehicle 
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SA  score  (object  and  location  correct) 


Results:  score  on  SA  task  (memorising  objects) 


wheelchair  ROV  stepping  around 

wheelchair  goggles  ROV  HMD 

condition 


ROV  joystick 
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p  <  .05 


wheelchair  ROV  stepping  around  ROV  joystick 

wheelchair  goggles  ROV  HMD 


condition 


rntwiS 


Results:  score  on  misery  scale 


Discussion 


In  general  the  data  show  that  the  ‘present’  conditions  are  better 
than  the  remote  control  conditions. 

There  are  no  effects  of  tele-presence  designs. 

There  was  a  high  (50%)  incidence  of  motion  sickness. 

The  interface  conditions  with  an  HMD  showed  a  large  increase  in 
MISC  rating.  This  confirms  the  provocative  nature  of  the  use  of 
an  HMD  as  the  primary  medium  for  orientation  in  a  virtual  or 
remote  world. 
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Conclusion 


We  can  conclude  that  the  present  exploratory  experiment  does 
not  provide  indications  that  support  the  hypotheses,  possibly 
because: 

The  resulting  interface  was  not  powerful  enough  to  result  in  a 
high  degree  of  presence.  This  may  be  related  to  the  technical 
implementation. 

The  disadvantage  (i.e.  the  risk  of  motion  sickness)  does  not 
outweigh  the  potential  advantages. 
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Follow-on 


Improved  the  technical  implementation. 

Added  artificial  information  such  as  vehicle  reference  points. 

•  Implemented  the  tele-presence  concept  in  a  wearable  device 
Expanding  the  sensory  suite  to  include  touch  information. 


Positive  responses  from  domain  experts. 
Data  analyses  of  follow-on 
experiment  in  progress. 
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